Project1 Bit by Bit 2.7

A, B, and C

1. Get the raw data from the Google Books NGram Viewer website (http://storage.googleapis.com/books/ngrams/books/datasetsv2.html). In particular, you should use version 2 of the English language corpus, which was released on July 1, 2012. Uncompressed, this file is 1.4 GB.
2. Recreate the main part of figure 3a of Michel et al. (2011). To recreate this figure, you will need two files: the one you downloaded in part (a) and the “total counts” file, which you can use to convert the raw counts into proportions. Note that the total counts file has a structure that may make it a bit hard to read in. Does version 2 of the NGram data produce similar results to those presented in Michel et al. (2011), which are based on version 1 data?
3. Now check your graph against the graph created by the NGram Viewer (https://books.google.com/ngrams).

Load the 1gram file

install.packages("tidyverse")

trying URL 'https://cran.rstudio.com/bin/macosx/el-capitan/contrib/3.5/tidyverse_1.2.1.tgz'
Content type 'application/x-gzip' length 88754 bytes (86 KB)
==================================================
downloaded 86 KB

The downloaded binary packages are in
    /var/folders/fb/p2xy06255856sb5hx3143jdr0000gn/T//RtmpNEl8cw/downloaded_packages

library(tidyverse)

── Attaching packages ─────────────────────────────────────────────────── tidyverse 1.2.1 ──
✔ ggplot2 3.1.0     ✔ purrr   0.2.5
✔ tibble  2.0.1     ✔ dplyr   0.7.8
✔ tidyr   0.8.2     ✔ stringr 1.4.0
✔ readr   1.3.1     ✔ forcats 0.3.0
── Conflicts ────────────────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()

setwd("~/Desktop/Project 1")
getwd()

[1] "/Users/KangBangg/Desktop/Project 1"

Filter by the three years I want: 1883, 1910, and 1950

ngram = ngram %>% rename(
  word = V1,
  year = V2,
  match_count = V3,
  volume_count = V4
  )
ngram1 = filter(ngram, word == "1883" | word == "1910" | word == "1950")
ngram1

Load the total count file

totalcount = read.table("totalcount.txt", sep = "\t", colClasses = "character")
totalcount

The current format of this is (year, match_count, page_count, volume_count). I have to make them into 4 columns. Use the separate function

totalcount = separate_rows(totalcount, V1:V425, convert = TRUE)
rownames(totalcount)[1] = "year"
rownames(totalcount)[2] = "match_count"
rownames(totalcount)[3] = "page_count"
rownames(totalcount)[4] = "volume_count"
finaltotalcount = as.data.frame(t(totalcount))
finaltotalcount

Take the match_count of ngram1 and divide it by the match_count of finaltotalcount to find the proportion of the ngram compared to total words used for that year.

figurea1 = select(ngram1,
                 word,
                 year,
                 match_count)
figurea2 = select(finaltotalcount,
                  year,
                  match_count)
figurea3 = left_join(figurea1, figurea2, by = "year") %>%
  mutate(frequency = match_count.x / match_count.y)
figurea3

Make the vizualization

ggplot(figurea3, aes(x = year, y = frequency, colour = word)) +
  geom_freqpoly(binwidth = 500, stat = "identity")

Ignoring unknown parameters: binwidth

D

4. Recreate figure 3a (main figure), but change the y-axis to be the raw mention count (not the rate of mentions).

Raw count vizualization

ggplot(figurea3, aes(x = year, y = match_count.x , colour = word)) +
  geom_freqpoly(binwidth = 500, stat = "identity")

Ignoring unknown parameters: binwidth

E

5. Does the difference between (b) and (d) lead you to reevaluate any of the results of Michel et al. (2011).Why or why not?

The way Michel used frequency to determine word use decay makes sense as overtime more words will be added to the corpus and the amount of word mentions over time could increase due to possible reasons as more people, more authors, better document collection, and more outlets to use such as social media. If Michel and other were to rely on the raw mention count overtime, he may of thought word use increases over time, but that is not the case.

F

6. Now, using the proportion of mentions, replicate the inset of figure 3a. That is, for each year between 1875 and 1975, calculate the half-life of that year. The half-life is defined to be the number of years that pass before the proportion of mentions reaches half its peak value. Note that Michel et al. (2011) do something more complicated to estimate the halflife— see section III.6 of their Supporting Online Information—but they claim that both approaches produce similar results. Does version 2 of the NGram data produce similar results to those presented in Michel et al. (2011), which are based on version 1 data? (Hint: Don’t be surprised if it doesn’t.)

Half-life calculation

ngram$word = as.integer(ngram$word)
hlNgram = filter(ngram, word >= 1875)
hlNgramf = filter(hlNgram, word <= 1975)
originalhl = left_join(hlNgramf, figurea2, by = "year") %>%
  mutate(frequency = match_count.x / match_count.y) %>%
  select(word, year, frequency) %>%
  mutate(yearG = year > word) %>%
  filter(yearG == TRUE)
hl = originalhl

After find the max frequency for each year and then filter frequency by 1/2 the max. Afterwards subtract the 1/2 frequency by the max.

hlM = originalhl %>% group_by(word) %>% summarise(frequency = max(frequency))
hlY = originalhl %>% group_by(year) 
hlT = left_join(hlM, hlY, by = "frequency", suffix = c(".x", ".y"))
hlT = select(hlT, word.x, year, frequency)
hlT = hlT %>% group_by(word.x) %>% filter (! duplicated(word.x)) %>%
  rename(word = word.x)
hlT2 = hlT %>% select(frequency) %>%
  left_join(hl, hlT, by = "word") %>%
  filter(frequency.x > (frequency.y / 2)) %>%
  select(word, frequency.x, year)


hlF = left_join(hlT, hlT2, by = "word") %>%
  filter(year.y > year.x) %>%
  group_by(word) %>% summarise(year.y = min(year.y))
hlT = hlT %>% select(word, year)
hlF = left_join(hlF, hlT, by = "word") 
hlF = mutate(hlF, half_life = (year.y - year))
hlF = hlF %>% rename(ngram_year = word)
hlF = hlF %>% select(ngram_year, half_life)
hlF

Make the half-life vizualization.

Does version 2 of the NGram data produce similar results to those presented in Michel et al. (2011), which are based on version 1 data? (Hint: Don’t be surprised if it doesn’t.) It does not as in Michel et al. their half life values for early years of the range 1875-1975 start off extremely high at 30 and then curve down later in the year range to an average of 15 years for half-life.

G

7. Were there any years that were outliers, such as years that were forgotten particularly quickly or particularly slowly? Briefly speculate about possible reasons for that pattern and explain how you identified the outliers. Any years with a half-life > 10 would be an outlier and I determined that by using geom_smooth which applies a conditional mean line onto the ggplot. The most likely explanation for these outliers is historical events that happened such as wars or tragedies. For the year 1912 it had a half-life of 26, looking at the events of that year, there was the tragic event of the Titanic sinking. However what is strange is if my guess were to be true, that would mean the year 1914 which is the start of WW1, would have a high half-life. Looking at it, it does not and looking at the end year of WW1, 1918, it has a half-life of 1 with a peak rate of mentions in 1921.

H

8. Now replicate this result for version 2 of the NGrams data in Chinese, French, German, Hebrew, Italian, Russian and Spanish.

Chinese

French

German

Hebrew

Italian

Russian

Spanish

I

1. Comparing across all languages, were there any years that were outliers, such as years that were forgotten particularly quickly or particularly slowly? Briefly speculate about possible reasons for that pattern.

Across all languages, there are outliers within the middle of the range of 1875-1975, specifically the 1gram 1925, my speculation would be that these are years with international events that were recorded in each language and there would be a high frequency of them with a high half-life due to the importance of those events that happened within those years. Looking at Hebrew, Chinese and Italian, they both have their largest outlier in 2 years close to each other which could be a shared event among them. China has a higher variance around its 1gram half-life frequency out of every other language between 1875-1925 which could be due to sociopolitical events within mandarin/cantonese speaking areas.